Method and apparatus for representing and searching for an object in an image

ABSTRACT

A method representing an object appearing in still or video image for use in searching, wherein the object appears in the image with a first two-dimensional outline, by processing signals corresponding to the image, comprises deriving a view descriptor of the first outline of the object and deriving at least one additional view descriptor of the outline of the object in a different view, and associating the two or more view descriptors to form an object descriptor.

This application is a Divisional of U.S. application Ser. No. 11/777,135filed Jul. 12, 2007, which is a Divisional of Prior application Ser. No.10/220,636 which is a national phase under 35 U.S.C. §371 of PCTInternational Application No. PCT/GB01/03044, which has an Internationalfiling date of Jul. 6, 2001, which designated the United States ofAmerica.

The present invention relates to a method and apparatus for representingan object in an image, especially for use in searching, for example, ina multimedia database. The invention also relates to a method andapparatus for searching for an object in an image.

It is known to store still or video images, for example in imagelibraries, using representations of objects appearing in images. Therepresentations are used in searching methods to enable imagescontaining objects of interest to be retrieved, The representations maybe based on various features of the objects, including colour, textureand shape.

Various methods for representing shapes of objects in images are known.Known methods include chain coding, the quad-tree method and thecurvature scale space representation method.

To perform a search in a image searching system, a user inputs a queryby presenting the system with a sketch or image of the object sought, orby selecting a view of an object stored on the system. The system thenderives or obtains a representation of the query object and compares thequery representation with the representations of images stored in thedatabase using an appropriate matching method, The closest matches aredisplayed on a display unit.

In most cases, an object as it appears in a video image is a projectionof a 3-dimensional real object onto a 2-dimensional image plane. The2-dimensional shape or outline of an object, and the colours and textureof the visible object surface, appearing in an image thus depend onfactors such as the viewing position, viewing angle and camera andoptical system parameters. Objects will therefore have differentfeatures such as outline, shape, colour, texture etc associated withdifferent views.

Our co-pending patent application, PCT/GBOO/01662, the contents of whichare incorporated herein by reference, discloses a method and apparatusfor representing an object appearing in an image where representationsof an object in a plurality of different 2-dimensional views are derivedand associated to form an object representation. Preferably, therepresentations are of the shape of the object, but they may, forexample, be of colour or texture in different views.

When applying the above-mentioned method to an object within a stillimage or a movie etc., there will be several shape or other descriptorsassociated with an image to represent the object features associatedwith different views. However, typically any one of these views will bevisible within the original image. Some of the features may beassociated with the non-visible parts of the object and are only fordescribing the 3D properties of the object.

If one cannot determine which of the multiple views of the object arevisible and which are not, the search method will not be able to findexactly specified views, For example, a search for an image depicting acar, using a side view, will also find top and front views of cars.

The inventive idea is to use visibility flags which state whichdescriptors actually relate to visible features within the image ormovie to which the descriptors are linked.

Accordingly, the present invention provides a method of representing anobject appearing in an image, the method comprising deriving a pluralityof view descriptors of the object, each view descriptor corresponding toa different view of the object, and indicating when a view correspondsto a view of the object appearing in the image.

In the invention disclosed in. PCT/IGB00/101662 shape descriptors areassociated with objects present within a piece of multi-media content.Such descriptors described the shape of the object from different views.These views can be either visible within the multi-media material ornot. The visibility flag will store this visibility information.

When the visibility data for each view is available the search method isable to filter out the objects with non-visible views when required.

An embodiment of the invention will be described with reference to theaccompanying drawings of which:

FIG. 1 is a block diagram of a system according to an embodiment of theinvention;

FIG. 2 is a diagram illustrating representation of an object in a videosequence;

FIG. 3 is a diagram illustrating representation of an object in a stillimage;

FIG. 4 is a diagram illustrating results of a first search method;

FIG. 5 is a diagram illustrating results of a second search method.

A system according to an embodiment of the invention is shown in FIG. 1.The system includes a control unit 2 such as a computer for controllingoperation of the system, a display unit 4 such as a monitor, connectedto the control unit 2, for displaying outputs including images and text,and a pointing device 6 such as a mouse for inputting instructions tothe control unit 2. The system also includes an image database 8 storingdigital versions of a plurality of still images and groups of images inthe form of video sequences and a descriptor database 10 storingdescriptor information, described in more detail below, for objectsappearing in each of the still images and video sequences stored in theimage database 8. Each of the image database 8 and the descriptordatabase is connected to the control unit 2.

In this embodiment, the elements of the system are provided on a singlesite, such as an image library, where the components of the system arepermanently linked.

A method of deriving a descriptor for an object according to anembodiment of the invention will be described below. The method will bedescribed first for an object having a relatively simple shape, in thiscase a cylindrical object.

In this example, in a video sequence stored in the image database 8, thecylindrical object appears twice. Referring to FIG. 2, in the firstappearance, the object outline corresponds to view 1, that is a viewfrom the side and in the second appearance, the object outlinecorresponds to view 3, that is a perspective view from the side andabove.

The database indexer recognises that the two outlines are of the sameobject and that the 3-dimensional object corresponding to those outlinesis a cylinder. For each of view 1 and view 3, shape descriptors usingthe curvature scale space (CSS) representation are derived. Also, thedatabase indexer selects any additional views that are deemedrepresentative of the object, even if they do not appear in the videosequence under consideration. In this example, view 2, which is a viewof a cylinder from above, is selected as representative. A CSSrepresentation of that view is also obtained.

The paper “Robust and Efficient Shape Indexing through Curvature ScaleSpace” by Farzin Mokhtarian, Sadegh Abbassi and Josef Kittler, Proc.British Machine Vision Conference, pp. 53-62, Edinburgh, UK 1996describes a method of curvature scale space representation and isincorporated herein by reference. Briefly, a curve representing theoutline of a shape is evolved by smoothing the curve. The curve isconsidered at a number of different stages in the evolution. Morespecifically, the curvature zero crossings are identified in a curvaturefunction at each of the stages of the evolution. A graph of thecurvature zero crossings is obtained by combining zero crossing from allstages of the evolution. One axis of the graph corresponds to σ whichrepresents the evolution parameter of the curve and the other axiscorresponds to a curve arc length parameter u. A shape is thenrepresented by the locations of the maxima of the contours in the graph.

For each view, a view descriptor is formed, including the appropriateCSS representation. The view descriptor may include other information,such as a global parameter, as described in PCT/GBO0/01662, orcolour/texture properties of the corresponding visible surfaces.

The view shape descriptors are combined to form a 3D object shapedescriptor. In addition, visibility flags indicating whether or not agiven view is visible in at least one frame in the video sequence areassociated with the object descriptor. In this example, the visibilityflag forms part of each view descriptor. Thus, here the view descriptorof each of view 1 and view 3 includes a visibility flag ‘1’ indicatingthat the view is visible in a frame of the video, and the viewdescriptor of view 2 has a visibility flag ‘0’, indicating that the viewis not visible in any frame.

3D object shape descriptors are obtained in the above manner for allobjects of interest appearing in images or groups of images in the imagedatabase 8.

As another example, referring to FIG. 3, three views are derived for acar appearing in a photograph. Corresponding view descriptors, includinga CSS representation, are obtained as described above for the caroutline in each view. The first view corresponds to the view in thephotograph and thus the associated view descriptor has visibility flag‘1’, indicating a visible view. The view descriptors for the other twoviews have visibility flags ‘0’, indicating that they correspond toviews which are not shown in the photograph. In this example, thevisibility flags are determined when extracting the outlines from thephotograph. In other words, if a shape is extracted directly from thephotograph, then it is given a visibility flag of ‘1’, but if the shapeis chosen by the indexer, then it has a visibility flag of ‘0’.Alternatively, the visibility flags can be added later manually byinspection.

Each view descriptor also has a reference pointer indicating in whichimage or which frame in which video sequence it appears, and also whichobject in the image or frame it is, for example, a view of an object mayhave a pointer indicating that it appears in frame 1000 of video 181 andit is object number 3.

The descriptor database 10 stores the 3D object shape descriptors forthe objects in images and video sequences stored in the image database8.

In the embodiment described above, for an object appearing in a videosequence, one set of view descriptors are derived for the entiresequence, and the view descriptors are used to indicate whether or notthe associated view appears in at least one frame of the sequence. In analternative embodiment, each frame is treated separately. In moredetail, a set of view descriptors are derived for an object appearing inthe frame, and the visibility flags indicate whether or not theassociated view appears in that frame, irrespective or whether or not itappears in another frame of the sequence. In another alternativeembodiment, one set of view descriptors are derived for an objectappearing in the video sequence, and a set of corresponding visibilityflags are derived and stored for each frame.

The visibility flags are used in a method of searching for an object inan image, as described below.

The user initiates a search by inputting a query. The query is input byusing the pointing device 6 to draw one or more shape outlines on thedisplay unit 4. The control unit 2 then derives the CSS view descriptorfor the query view. Alternatively, the user may input a query byselecting a query shape or shapes from a menu of shapes displayed on thedisplay unit 4 by the control unit 2. In such a case, the viewdescriptors may already be available in the descriptor database 10. Theuser also indicates, by checking a box, whether or not the search shouldbe limited to views which are the same as the query view. If the searchis not so limited, then a search and matching procedure is performedessentially as described in PCTIGB00/01662, by determining thesimilarity between the query object descriptor and the objectdescriptors stored in the descriptor database 10, hereinafter describedas model object descriptors.

Briefly, each query view descriptor is compared with each model viewdescriptor for the model object descriptor under consideration in acomparator using a matching function to derive view-similarity measures.A comparison of an ith query view descriptor and the jth model viewdescriptor results in a view similarity measure S_(i,j). When there ismore than one query view, the median of the view similarity measures istaken as an object similarity measure.

The matching procedure results in n similarity measures S, where n isthe number of 3D object descriptors in the descriptor database. The nsimilarity measures are then ordered starting from the lowest valuewhich indicates the closest match. Them lowest values, where m is avalue selected by the user or determined by the set-up of the controlunit, are then selected, and the corresponding m images, (some of whichmay be an image from a video sequence) including the object is displayedon the display unit 4.

If the search is limited to views which are the same as the query view,then non-visible model views are filtered out during the searchingprocess. This can be done by matching the query view descriptor usingthe matching function only with the model view descriptors whichindicate that the corresponding view is visible in an image. In otherwords, when calculating the similarity measure using the matchingfunction, model view descriptors having visibility flag ‘0’ are omitted,Alternatively, view similarity measures may be calculated for all modelview descriptors, but results for model view descriptors for which thevisibility flag is zero are then omitted from further processing.Alternatively, the visibility flag can be considered when selecting thesearch results for display. When displaying the results of the matchingprocedure, in the case of a video sequence, the frame containing thematched view is displayed.

FIG. 4 shows an example of how the multiple views searching methoddisclosed in PCT/GS00/01662 finds and returns results which are of thesame shape but are not of the same view as the query item. FIG. 5 on theother hand shows how the visibility flag within each of the viewdescriptors can help return results which are of the same shape and thesame view as the query item.

A system according to the invention may, for example, be provided in animage library. Alternatively, the databases may be sited remote from thecontrol unit of the system, connected to the control unit by a temporarylink such as a telephone line or by a network such as the internet. Theimage and descriptor databases may be provided, for example, inpermanent storage or on portable data storage media such as CD-ROMs orDVDs.

Components of the system as described such as the selectors andcomparators may be provided in software or hardware form. Although theinvention has been described in the form of a computer system, it couldbe implemented in other forms, for example using a dedicated chip.

Specific examples have been given of methods of representing a 2D shapeof an object and of methods for calculating values representingsimilarities between two shapes but any suitable such methods can beused.

The various views of an object making up the complete or quasi-completeshape description may, for example, be provided by the film producer.For example, in a film including a car, the producer may arrange for 20different views of the car to be shot for use in indexing the film inthe database. Alternatively, a link may be provided between all thedifferent views of an object in a sequence to enable the views to befound, and the shape for any useful views not appearing in the sequencemay, for example, be provided when data for the first view of the objectis included.

The invention can also be used, for example, for matching images ofobjects for verification purposes, or for filtering.

The invention is applicable to single images, and images in a sequenceof images, such as images from a film or a video, or a collection ofimages that are associated in some way, such as being on the same webpage.

In the embodiments described above, view descriptors are derived fromthe shape of the object in different views. The view descriptors can bederived from other features of the object, such as the colour ortexture, either instead of or as well as from shape. Various features ofthe object, such as shape, colour, texture, can form the basis of theview descriptors alone or in combination, and view descriptors can bebased on different features for different views.

1. A method of representing an object appearing in an image using one ormore processors, the method comprising: deriving, using a processor, aplurality of view descriptors of the object, each view descriptorcorresponding to a different view of the object; indicating for eachview descriptor whether or not the respective view corresponds to a viewof the object appearing in the image by associating, using a processor,a visibility flag with each view descriptor, the visibility flagindicating whether or not the view corresponds to a view of the objectappearing in the image, wherein at least one view descriptor comprises arepresentation of the texture of the object in the respective view,wherein each view is a view of the object from a respective angle ofview, the plurality of view descriptors including at least onedescriptor of the view of the object in the image and at least onedescriptor of a view of the object from an angle of view different fromthe angle of view of the object in the image.
 2. A method as claimed inclaim 1 comprising associating the view descriptors to form an objectdescriptor.
 3. A method as claimed in claim 1 which is for indexing theimage or collection of images for searching purposes.
 4. A method ofsearching for a query object in stored images and groups of images, themethod comprising inputting a query, matching the query withrepresentations of the stored images derived using a method as claimedin claim 1, and retrieving the closest matches, wherein the query inputincludes indicating whether or not the search should be limited toimages or groups of images which show the query object in the same viewas the query view.
 5. An apparatus an image database storing stillimages or groups of images; a descriptor database storing descriptorsfor objects appearing in the images stored in the image database; and aprocessor configured to perform the method as claimed in claim
 1. 6. Acomputer-readable storage medium storing computer-executable processsteps, for implementing a method as claimed in claim
 1. 7. A computersystem comprising: an image database storing still image or groups ofimages; a descriptor database storing descriptors for objects appearingin the images stored in the image database; and a control unitprogrammed to perform the method as claimed in claim
 1. 8. A method asclaimed in claim 1, wherein said visibility flag is a first value if theview corresponds to a view of the object appearing in the image and asecond value if the view corresponds to a view of the object notappearing in the image.
 9. A method of representing an object appearingin a group of images using one or more processors, the methodcomprising: deriving, using a processor, a plurality of view descriptorsof the object, each view descriptor corresponding to a different view ofthe object; indicating for each view descriptor whether or not therespective view corresponds to a view of the object appearing in thegroup of images, by associating, using a processor, a visibility flagwith each view descriptor, the visibility flag indicating whether or notthe view corresponding to a view of the object appearing in the group ofimages, wherein at least one view descriptor comprises a representationof the texture of the object in the respective view, wherein each viewis a view of the object from a respective angle of view, the pluralityof view descriptors including at least one descriptor of the view of theobject in the image and at least one descriptor of a view of the objectfrom an angle of view different from the angle of view of the object inthe image.
 10. A method as claimed in claim 9 comprising indicating, foreach image in the group of images, for each view descriptor whether ornot the respective view corresponds to a view of the object appearing insaid image.
 11. A method as claimed in claim 9 wherein the group ofimages is a sequence of images.
 12. A method as claimed in claim 11wherein the sequence of images is from a video or film.
 13. A method asclaimed in claim 9, wherein said visibility flag is a first value if theview corresponds to a view of the object appearing in the group ofimages and a second value if the view corresponds to a view of theobject not appearing in the group of images.